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HIGHLEVEL ACTIVE PEN MATRIX 

Cross-Reference to Related Applications 

5 The present application is based on U.S. provisional patent application no. 60/247,400, 

filed November 10, 2000, entitled "Highlevel Active Pen Matrix". 

The present application is also related to application Serial No. (Atty docket 

3797.00066), entitled "Method and Apparatus For Improving the Appearance of Digitally 

Represented Handwriting", filed November 10, 2000; to application Serial No. (Atty docket 
10 3797.00069), entitled "Selection Handles in Editing Electronic Documents", and filed November 

10, 2000; to application Serial No. (Atty docket 3797.00070), entitled "Insertion Point Bungee 

Space Tool", and filed November 10, 2000; to application Serial No. (Atty docket 3797.00072), 
"jS entitled "Simulating Gestures of a Mouse Using a Stylus and Providing Feedback Thereto", and 
[ j filed November 10, 2000; to application Serial No. (Atty docket 3797.00074), entitled "System 
|5 and Method For Accepting Disparate Types Of User Input", and filed November 10, 2000; to 
r|] application Serial No. (Atty docket 3797.00075), entitled "In Air Gestures", and filed 
10 concurrently with the present application; to application Serial No. (Atty docket 3797.00076), 
u entitled "Mouse Input Panel Windows Class List", and filed November 10, 2000; to application 

Serial No. (Atty docket 3797.00077), entitled "Mouse Input Panel and User Interface", and filed 
20 November 10, 2000; and to application Serial No. (Atty docket 3797.00079), entitled "System 

and Method For Inserting Implicit Page Breaks", and filed November 10, 2000; each of which is 

incorporated by reference herein. 
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Field of the Invention 

Aspects of the present invention are directed generally to apparatus and methods for 
controlling a graphical user interface (GUI). More particularly, the present invention relates to 
receiving user input, determining based on the user input what the user wants to do, and 
5 performing a function related to the desired input. 
Background of the Invention 

Typical computer systems, especially computer systems using graphical user interface 
(GUI) systems such as Microsoft WINDOWS, are optimized for accepting user input from one 
or more discrete input devices such as a keyboard and for entering text, and a pointing device 

10 such as a mouse with one or more buttons for driving the user interface. Virtually all software 
applications designed to run on Microsoft WINDOWS are optimized to accept user input in the 

;ij same manner. For instance, many applications make extensive use of the right mouse button (a 

I j "right click") to display context-sensitive command menus. The user may generate other 
gestures using the mouse such as by clicking the left button of the mouse (a "left click"), or by 

f § clicking the left or right button of the mouse and moving the mouse while the button is depressed 

Ifl (either a "left click drag" or a "right click drag"). 

^ In some environments, a mouse is not usable or desirable. For example, in a digitizer 

tablet environment, the primary input device may be a stylus. While a stylus attempts to provide 
pad and paper-like feel to a computing environment, current systems are limited. For example, 

20 the use of a stylus in a graphical user interface is limited to tapping on various items for 
selection. See, for example, the Palm-series of products using the Palm OS 3.0 operating system. 
Further, in stylus-based input environments, a user is continually forced to select tools or 
operations from a remote tool bar, generally on a top or bottom of a screen. While a user can 
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type in letters or have the digitizer recognize handwriting, these operations require selecting a 
keyboard input mode and writing in a predefined portion of the digitizer, respectively. In short, 
requiring a user to tell the computer, for every new input, what a user wants to do makes stylus- 
based computing difficult for the average user. Accordingly, stylus based inputs have been 
5 relegated to personal data assistants (PDAs) where significant user input is not possible. 
Mainstream computing still requires the use of at least a keyboard and mouse (or mouse -based 
input device, for example, trackballs, touch-pads, and other mouse substitutes). 

Accordingly, a need exists for permitting a user to perform all operations of a mouse-type 
device using a stylus. 
10 Summary of the Invention 

q As discussed in the various copending patent applications incorporated herein by 

S reference, aspects of the present invention are directed to a tablet-like computer that allows users 
Sj to directly write on a display surface using a stylus. The display surface may physically, 
■ optically, and or electro magnetically detect the stylus. The computer may allow the user to write 
lg and to edit, manipulate, and create objects through the use of the stylus. Many of the features 
discussed in these copending applications are more easily performed by use of the various 
== ; aspects of the present invention discussed herein. 

An aspect of the present invention is directed to methods and apparatus for simulating 
gestures of a mouse by use of a stylus on a display surface. The present invention determines the 
20 operation a user wants to perform based on the user's input. This determination may include 
reference to other information including the location of the user's input on a digitizer (e.g., 
location on a screen) and the status of other objects or elements as displayed. By using this 
information, the system determines what the user wants to do and implements the action. 
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A number of inputs with a stylus are possible. For example, a user may tap a stylus, 
stroke the stylus, hold the stylus at a given point, or hold then drag the stylus. Other inputs and 
combinations are possible as noted by the above-identified applications, which are expressly 
incorporated herein by reference. 

As to a stroke operation, the system may drag an object, may maintain a current state or 
operation, or being inking. Inking may include writing, drawing, or adding annotations as 
described in greater detail in U.S. Serial No. 60/212,825, filed June 21, 2000, entitled "Methods 
for Classifying, Anchoring, and Transforming Ink Annotations" and incorporated by reference. 

As to a tap operation, the system may add to existing writing, may select a new object, 
insert a cursor or insertion point, or may perform an action on a selected object. 

As to a hold operation, the system may simulate a right mouse button click or other 
definable event. 

As to a hold and drag operation, the system may drag a selected object or perform other 
functions. 

These and other features of the invention will be apparent upon consideration of the 
following detailed description of preferred embodiments. Although the invention has been 
defined using the appended claims, these claims are exemplary in that the invention is intended 
to include the elements and steps described herein in any combination or subcombination. 
Accordingly, there are any number of alternative combinations for defining the invention, which 
incorporate one or more elements from the specification, including the description, claims, and 
drawings, in various combinations or subcombinations. It will be apparent to those skilled in the 
relevant technology, in light of the present specification, that alternate combinations of aspects of 
the invention, either alone or in combination with one or more elements or steps defined herein, 
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may be utilized as modifications or alterations of the invention or as part of the invention. It is 
intended that the written description of the invention contained herein covers all such 
modifications and alterations. 
Brief Description of the Drawings 

The foregoing summary of the invention, as well as the following detailed description of 
preferred embodiments, is better understood when read in conjunction with the accompanying 
drawings, which are included by way of example, and not by way of limitation with regard to the 
claimed invention. In the accompanying drawings, elements are labeled with three-digit 
reference numbers, wherein the first digit of a reference number indicates the drawing number in 
which the element is first illustrated. The same reference number in different drawings refers to 
the same element. 

Figure 1 is a schematic diagram of a general-purpose digital computing environment that 
can be used to implement various aspects of the invention. 

Figure 2 is a plan view of a tablet computer and stylus that can be used in accordance 
with various aspects of the present invention. 

Figures 3-7 are flowcharts showing a variety of steps for interpreting a user's input in 
accordance with embodiments of the present invention. 
Detailed Description of Preferred Embodiments 

The present invention may be more readily described with reference to Figures 1-7. 
Figure 1 illustrates a schematic diagram of a conventional general-purpose digital computing 
environment that can be used to implement various aspects of the present invention. In Figure 
1, a computer 100 includes a processing unit 110, a system memory 120, and a system bus 130 
that couples various system components including the system memory to the processing unit 
1 10. The system bus 130 may be any of several types of bus structures including a memory bus 
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or memory controller, a peripheral bus, and a local bus using any of a variety of bus 
architectures. The system memory 120 includes read only memory (ROM) 140 and random 
access memory (RAM) 150, 

A basic input/output system 160 (BIOS), containing the basic routines that help to 
5 transfer information between elements within the computer 100, such as during start-up, is stored 
in the ROM 140. The computer 100 also includes a hard disk drive 170 for reading from and 
writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a 
removable magnetic disk 190, and an optical disk drive 191 for reading from or writing to a 
removable optical disk 192 such as a CD ROM or other optical media. The hard disk drive 170, 
l£k s magnetic disk drive 180, and optical disk drive 191 are connected to the system bus 130 by a 
C] hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive 
interface 194, respectively. The drives and their associated computer-readable media provide 
%j nonvolatile storage of computer readable instructions, data structures, program modules and 
f other data for the personal computer 100. It will be appreciated by those skilled in the ait that 
Ipp other types of computer readable media that can store data that is accessible by a computer, such 
z! as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random 
access memories (RAMs), read only memories (ROMs), and the like, may also be used in the 
example operating environment. 

A number of program modules can be stored on the hard disk drive 170, magnetic disk 
20 190, optical disk 192, ROM 140 or RAM 150, including an operating system 195, one or more 
application programs 196, other program modules 197, and program data 198. A user can enter 
commands and information into the computer 100 through input devices such as a keyboard 101 
and pointing device 102. Other input devices (not shown) may include a microphone, joystick, 
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game pad, satellite dish, scanner or the like. These and other input devices are often connected 
to the processing unit 110 through a serial port interface 106 that is coupled to the system bus, 
but may be connected by other interfaces, such as a parallel port, game port or a universal serial 
bus (USB). Further still, these devices may be coupled directly to the system bus 130 via an 

5 appropriate interface (not shown). A monitor 107 or other type of display device is also 
connected to the system bus 130 via an interface, such as a video adapter 108. In addition to the 
monitor, personal computers typically include other peripheral output devices (not shown), such 
as speakers and printers. In a preferred embodiment, a pen digitizer 165 and accompanying pen 
or stylus 166 are provided in order to digitally capture freehand input. Although a direct 
10,. connection between the pen digitizer 165 and the processing unit 110 is shown, in practice, the 

./j pen digitizer 165 may be coupled to the processing unit 110 via a serial port, parallel port or 

r>i other interface and the system bus 130 as known in the art. Furthermore, although the digitizer 
165 is shown apart from the monitor 107, it is preferred that the usable input area of the digitizer 
165 be co-extensive with the display area of the monitor 107. Further still, the digitizer 165 may 
llEM be integrated in the monitor 107, or may exist as a separate device overlaying or otherwise 

i; appended to the monitor 107. 

The computer 100 can operate in a networked environment using logical connections to 
one or more remote computers, such as a remote computer 109. The remote computer 109 can 
be a server, a router, a network PC, a peer device or other common network node, and typically 
20 includes many or all of the elements described above relative to the computer 100, although only 
a memory storage device 111 has been illustrated in Figure 1. The logical connections depicted 
in Figure 1 include a local area network (LAN) 112 and a wide area network (WAN) 113. Such 
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networking environments are commonplace in offices, enterprise-wide computer networks, 
intranets and the Internet. 

When used in a LAN networking environment, the computer 100 is connected to the 
local network 112 through a network interface or adapter 114. When used in a WAN networking 
environment, the personal computer 100 typically includes a modem 115 or other means for 
establishing a communications over the wide area network 113, such as the Internet. The 
modem 115, which may be internal or external, is connected to the system bus 130 via the serial 
port interface 106. In a networked environment, program modules depicted relative to the 
personal computer 100, or portions thereof, may be stored in the remote memory storage device. 

It will be appreciated that the network connections shown are exemplary and other 
techniques for establishing a communications link between the computers can be used. The 
existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the 
like is presumed, and the system can be operated in a client-server configuration to permit a user 
to retrieve web pages from a web-based server. Any of various conventional web browsers can 
be used to display and manipulate data on web pages. 

Figure 2 illustrates a tablet PC 201 that can be used in accordance with various aspects of 
the present invention. Any or all of the features, subsystems, and functions in the system of 
Figurel can be included in the computer of Figure 2. Tablet PC 201 includes a large display 
surface 202, e.g., a digitizing flat panel display, preferably, a liquid crystal display (LCD) screen, 
on which a plurality of windows 203 is displayed. Using stylus 204, a user can select, highlight, 
and write on the digitizing display area. Examples of suitable digitizing display panels include 
electromagnetic pen digitizers, such as the Mutoh or Wacom pen digitizers. Other types of pen 
digitizers, e.g., optical digitizers, may also be used. Tablet PC 201 interprets marks made using 
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stylus 204 in order to manipulate data, enter text, and execute conventional computer application 
tasks such as spreadsheets, word processing programs, and the like. 

A stylus could be equipped with buttons or other features to augment its selection 
capabilities. In one embodiment, a stylus could be implemented as a "pencil" or "pen", in which 
5 one end constitutes a writing portion and the other end constitutes an "eraser" end, and which, 
when moved across the display, indicates portions of the display are to be erased. Other types of 
input devices, such as a mouse, trackball, or the like could be used. Additionally, a user's own 
finger could be used for selecting or indicating portions of the displayed image on a touch- 
sensitive or proximity-sensitive display. Consequently, the term "user input device", as used 
1ft herein, is intended to have a broad definition and encompasses many variations on well-known 
sj input devices. 

ffi Region 205 shows a feed back region or contact region permitting the user to determine 

y where the stylus as contacted the digitizer. In another embodiment, the region 205 provides 
f visual feedback when the hold status of the present invention has been reached. 
Il5f Figures 3-7 show various flowcharts for determining what a user wants to do based on a 

74 user's interaction with the digitizer. As will be discussed below, the user contacts the digitizer 
where the user wants to begin writing, tapping, annotating, dragging, etc. In the case where the 
digitizer is superimposed over a display, the user's contact with the digitizer is directed at 
operating at (or near) the contact point between the user's stylus and the currently displayed 
20 information at or near the contact point. 

In step 301, the system senses a contact or other indication of an action. In one 
embodiment the contact may be the stylus contacting the surface of the digitizer. In another 
embodiment, the action may be bringing the tip of the stylus near the digitizer's surface. Further, 
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if the stylus includes another signaling method (for example, a radio transmitter transmitting a 
signal to the digitizer signaling a user's input), the digitizer (or related input mechanism or 
mechanisms) interpret the received signal as a user's input. Other methods of starting an 
operation or writing or contact with a digitizer are known in the art. For purposes of illustration 
and description, the system and method reference physical contact with the digitizer. All other 
ways of providing signals to a processor are considered within the scope of the invention and are 
not mentioned here for simplicity. 

In step 302, the system determines the contact position and what lies beneath the contact 
position (for example, an object, a drawing, blank space, ink, and the like). In step 303, the 
system determines if the stylus has moved beyond a first threshold (time, distance, rate, or 
acceleration, and the like). In one embodiment, the threshold is set to the minimum resolvable 
movement. In another embodiment, the threshold is set higher to account for shaky hands, 
vibrations of the digitizer or tablet pc (for example, if trying to use the system while driving in a 
car over a bumpy road). It is noted that objects may have all the same threshold. Alternatively, 
objects may have different thresholds. This may be dependent on the object, the size of the 
object, the state of the system, the state of the object, and the like. 

If the first threshold has been exceeded, then the system proceeds to step 304 where the 
user's input is classified as a stroke and the system steps to point A 305. If the first threshold has 
not been exceeded, the system determines if the stylus was still in contact with the digitizer when 
a time threshold had expired in step 306. If no (meaning that the stylus was still in contact with 
the digitizer surface), the system classifies the input as a tap in step 307 and proceeds to point B 
308. 
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If the stylus was still in contact with the surface after the time threshold in step 306, the 
system determines if a second move threshold was exceeded in step 309. The first and second 
move thresholds may be identical or different. For example, both may be .25 mm. Or, the first 
may be .5 mm or one mm and the second be .3 mm. Further, the first may be 1.2 mm or more 
and the second may be .5mm or more. In short, any values may be used as long as they are not 
obtrusive to the user. The second threshold may be determined only after the time threshold of 
step 306 has expired. In this example, the second threshold may be higher than the first threshold 
(or it may be the same or smaller). 

If the second move threshold was not exceeded, then the system classifies the input as a 
hold in step 310 and proceeds to point C 311. If the second move threshold was exceeded, then 
the system classifies the input as a 'hold and drag' in step 312 and moves to point D 313. 

Figure 4 shows point A as starting point 401. Here, the system classified the input as a 
stroke and begins stroke processing in step 402. In step 403, the system determines if the stroke 
started on a draggable object. If yes, the system determines in step 404 whether drag threshold 
was exceeded (for example, .25 inches, .25 inches per second and the like). If so, the system 
classifies the stroke as a drag in step 405 and performs a function that is dependent on the object. 
For example, the drag may extend a selection as described in greater detail in "Selection Handles 
in Editing Electronic Documents," filed concurrently with the present application (attorney 
docket 03797.00069), and expressly incorporated by reference. Also, the drag may operate a 
bungee tool as described in Serial No. (Atty docket 3797.00070), entitled "Insertion Point 
Bungee Space Tool", and filed concurrently with the present application, and expressly 
incorporated herein. 



11 



Banner & Witcoff, Ltd. 
03797.00086 / 163138.2 

If, in step 404, the drag threshold has not been exceeded, the system maintains the current 
state (with the object being selected or not) in step 407. If the stroke was not over a draggable 
object in step 403, the system determines if the area under the contact point is inkable in step 
408. For example, inkable may mean an area capable of receiving ink (including drawings, 
annotations, or writing) as detailed in serial no. 60/212,825, filed June 21, 2000, and expressly 
incorporated herein by reference for essential subject matter. By contrast, a control button (for 
copy, save, open, etc.) may not be inkable. If inkable in step 408, the system permits inking 
(drawing, writing, annotating and other related functions) in step 409. If not inkable, the system 
maintains the current state (objects selected or not) in step 407. 

In Figure 5 A, the system starts at point B 501 and operates on the input as a tap 502. The 
system determines whether the tap was on an area or object that is inkable in step 503. If yes, the 
system determines whether any ink was recently added or "wet" (for example, less than .5 or 1 
second old) in step 504. If so, the system considers the tap as a dot to be added to the ink in step 
505 (and adds the dot). If no wet ink exists, then the system determines if the tap was over a 
selectable object in step 506. It is noted that steps 503 and 504 may be combined. If the tap was 
over a selectable object, then the system determines if the object was already selected in step 
507. If it was not, then the system selects the tapped object in step 508. If a previous object had 
been selected, the system cancels the previous or old selection in step 509. If the object was 
previously selected as determined by step 507, the system performs an action relevant to the 
object in step 510. This action may include editing the object, performing a predefined operation 
(for example, enlarge, shrink and the like). From step 506, if the tap was not on a selectable 
object, then the system proceeds to point BB 512. 
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Figure 5B shows additional processing to Figure 5A. As point BB 512, the system 
determines if the tap was in a space between text (referred to herein as an inline space) in step 
513. If yes, the system places an insertion point at the tap point in step 514. As shown in a 
broken lined box, the system may also cancel any old or previous selections in step 515. If no, 
then the system determines if the tap point has ink nearby in step 518. If the system determines 
that the tap was nearby ink, then the system adds a dot to the ink in step 516. If there was an old 
selection, then the system cancels the old selection in step 517 (as shown by a broken line box). 

If not nearby ink in step 518, the system determines if the tap is on an active object in 
step 519. If the tap was not on an active object, the system places an insertion point at the tap 
point or performs some other definable action in step 520. Again, if there was an old selection, 
then the system cancels the old selection in step 521 (as shown by a broken line box). If the tap 
was on an active object as determined by step 519, the system performs an action in step 522. 
The action may be definable by the user or relate to any function desirable. In one embodiment, 
the action may be to perform a function to operate a selection handle or bungee space tool as 
described in "Selection Handles in Editing Electronic Documents," filed concurrently with the 
present application (attorney docket 03797.00069), and expressly incorporated by reference. 
Also, the drag may operate a bungee tool as described in Serial No. (Atty docket 3797.00070), 
entitled "Insertion Point Bungee Space Tool", and filed concurrently with the present 
application, and expressly incorporated herein. Other operations are known in the an. and 
incorporated herein. 

Figure 6 relates to holding a stylus beyond a time threshold. Starting from point C 601, 
the system classifies the user input as a hold operation in step 602. Next, the system simulates a 
right mouse button click or other definable event in step 603. The functions associated with step 
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603 are described in greater detail in U.S. application Serial No. (Atty docket 3797.00072), 
entitled "Simulating Gestures of a Mouse Using a Stylus and Providing Feedback Thereto", filed 
November 10, 2000, whose contents are expressly incorporated herein by reference. 

Figure 7 relates to holding a stylus beyond a time threshold and moving the stylus. 
Starting from point D 701, the system classifies the user input as a hold and drag operation in 
step 702. Next, in step 703 the system drags the selected object as directed by the user. 

There are a number of alternatives associated with dragging. If the hold and drag relates 
to an inline space, the system may use this hold and drag function to select text. Similarly, one 
may use this function to select a drawing encountered by the dragged stylus. Further, one may 
select both text and drawings in this manner. Also, the cursor's point may become a selection 
tool that leaves a trail behind it. In this regard, the user may loop a number of objects, drawing or 
text in this regard. The looping of the objects may result in the selecting of the objects. 

An alternate embodiment of the present invention relates to modifying ink drawings or 
annotations. For example, if one added an annotation (from step 409) to text, one may 
manipulate the text (for example, by inserting new text) and have the annotation track the 
manipulation of the text. So, if one circled text then added text to the circled text, the annotation 
would expand to include the added text as well. This is described in relation to in U.S. Serial No. 
60/212,825, filed June 21, 2000, entitled "Methods for Classifying, Anchoring, and 
Transforming Ink Annotations" and incorporated by reference. 

While exemplary systems and methods embodying the present invention are shown by 
way of example, it will be understood, of course, that the invention is not limited to these 
embodiments. Modifications may be made by those skilled in the art, particularly in light of the 
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foregoing teachings. For example, each of the elements of the aforementioned embodiments may 
be utilized alone or in combination with elements of the other embodiments. 



15 



